Geometry Attention Transformer with position-aware LSTMs for image captioning

In recent years, Transformer structures have been widely applied in image captioning with impressive performance. However, previous works often neglect the geometry and position relations of different visual objects. These are thought as crucial information for good results. Aiming to further promote by Transformers, this paper proposes an improved Geometry Attention (GAT) framework. order obtain geometric representation ability, two novel geometry-aware architectures designed respectively encoder decoder our GAT i) a gate-controlled self-attention refiner, ii) group position-LSTMs. The first one explicitly incorporates relative spatial into representations encoding steps, second precisely informs word positions generating caption texts. extracted pretrained Faster-RCNN network. Our ablation study has proved that these optimization modules could efficiently improve performance captioning. experiment comparisons on datasets MS COCO Flickr30K, also show outperform current state-of-the-art models.